Image pickup apparatus capable of applying color conversion to captured image and control method thereof

ABSTRACT

In an image sensing apparatus adapted to apply user-settable image processing to a sensed image, display in an electronic viewfinder of an image to which a user-set image processing is applied and an image to which such image processing is not applied is switched with each passage of a predetermined period of time or in response to a user operation. Implementing a preview function with the electronic viewfinder image enables the user to check the results of the image processing in real time prior to actual image sensing by comparing the processed image and the original image to provide an easy-to-use preview function.

This is a continuation of U.S. patent application Ser. No. 11/332,379,filed Jan. 17, 2006, which issued as U.S. Pat. No. 7,538,308 on May 26,2009.

FIELD OF THE INVENTION

The present invention relates to an image sensing apparatus and acontrol method thereof, and more particularly, to an image sensingapparatus and a control method thereof that enables image sensing withimage processing set by a user applied.

BACKGROUND OF THE INVENTION

In recent years, with the spread of personal computers and digitalcameras, image processing has come to be performed widely. Where imageprocessing is applied to an original image using an image processingapplication software, which is also called a photo retouch software, itis necessary to determine the values of parameters such as filterapplication range and intensity and the extent of color tone adjustment.

Determining these sorts of parameter values is difficult for a beginner,and therefore applications that have the ability to set the bestparameter values automatically are also available. However, with suchautomatic setting it is difficult to obtain the best value for each andevery image, and normally most such applications determine the finalparameter values through some sort of trial-and-error process.

In order to facilitate this sort of adjustment of parameter values basedon trial and error, an application having a so-called preview functionthat enables a plurality of processed images to be displayed in such away that the images can be compared is known.

By contrast, image sensing apparatuses equipped with simple imageprocessing capabilities are available as well. Specifically, functionsare known that perform such effects when sensing images as intensifyingcolors so as to create more vivid color images, or, conversely, creatingplain images, or again making the entire screen monochromatic orsepia-tinted.

In addition, it is desirable to have a configuration that enables thedisplay not only of a preview screen but also of the original imageafter displaying the preview screen, in order to permit the effects ofthe image processing to be checked and to allow the image processing tobe reset if necessary. Although such a configuration is of coursenecessary in such cases as, for example, conversion of the originalimages into a sepia-tinted image or a monochrome image, or where imageprocessing is applied to the image screen as a whole such as in edgeemphasis, it is even more necessary in the case of image processing thatis applied to only a portion of an image.

For example, when applying such image processing as converting aspecific color in the original image (for example, red) to another color(for example, yellow) the preview image changes the red area in theoriginal image to yellow. In such a case, without being able to comparethe preview image with the original image it is difficult for the userto check the effect of the image processing, and the user can find itdifficult to change the source color (in this case red) or thedestination color (yellow) because the original colors are lost.

SUMMARY OF THE INVENTION

The present invention is conceived in light of the problem with theconventional art described above, and has as its object to provide aneasy-to-use preview capability in an image sensing apparatus capable ofadapting user-settable image processing to image sensing.

According to an aspect of the present invention, there is provided animage sensing apparatus having an image-sensing unit and display unit,the apparatus adapted to use the display means as an electronicviewfinder by continuously displaying images obtained by theimage-sensing unit on the display means, the apparatus comprising: imageprocessing unit adapted to apply image processing set arbitrarily to animage displayed on the electronic viewfinder; and control unit adaptedto display on the electronic viewfinder one after the other an image towhich the image processing is applied by the image processing unit andan image to which the image processing is not applied by the imageprocessing unit.

According to another aspect of the present invention, there is provideda control method for an image sensing apparatus having an image-sensingunit and display unit, the apparatus adapted to use the display unit asan electronic viewfinder by continuously displaying images obtained bythe image-sensing unit on the display unit, the method comprising: imageprocessing step of applying image processing set arbitrarily to an imagedisplayed on the electronic viewfinder; and control step of displayingon the electronic viewfinder one after the other an image to which theimage processing is applied in the image processing step and an image towhich the image processing is not applied in the image processing step.

With the above-described configuration, the image sensing apparatus ofthe present invention enables a user to check the effect of an imageprocessing set by the user by displaying a preview of the processedimage prior to actual image sensing that applies the image processingset by the user. In addition, the image sensing apparatus of the presentinvention enables the user to more clearly check the effects of imageprocessing because the original image, which is not processed, is alsodisplayed together with the above-described processed image.

Other features and advantages of the present invention will be apparentfrom the following description taken in conjunction with theaccompanying drawings, in which like reference characters designate thesame or similar parts throughout the figures thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and,together with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram showing an example of the configuration of adigital camera as an image sensing apparatus according to one embodimentof the present invention;

FIG. 2 is a perspective view showing one external configuration of animage sensing apparatus according to one embodiment of the presentinvention;

FIG. 3 is a flow chart illustrating a sensing operation of the digitalcamera 100 according to the present embodiment in case that an effect isto be applied to a sensed image;

FIG. 4 is a diagram showing one example of a display on a colordesignation screen of the digital camera 100 according to one embodimentof the present invention;

FIG. 5 is a diagram showing another example of a display on the colordesignation screen of the digital camera 100 according to one embodimentof the present invention;

FIG. 6 is a diagram showing another example of a display on the colordesignation screen of the digital camera 100 according to one embodimentof the present invention;

FIG. 7 is a flow chart illustrating a preview process in the digitalcamera 100 according to a first embodiment of the present invention;

FIG. 8 is a diagram showing one example of a preview image in thedigital camera 100 according to the first embodiment of the presentinvention;

FIG. 9 is a diagram showing another example of a preview image in thedigital camera 100 according to the first embodiment of the presentinvention; and

FIG. 10 is a flow chart illustrating the preview process in the digitalcamera 100 according to a second embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Preferred embodiments of the present invention will now be described indetail in accordance with the accompanying drawings.

The embodiments described below use the example of the present inventionadapted to a digital still camera as one example of an image sensingapparatus 100. However, the present invention can be adapted not just toa digital still camera but to any image sensing apparatus that recordsimages as electrical signals or as digital data, such as a digital videocamera. Furthermore, the present invention can also be adapted to anyapparatus capable of using such an image sensing apparatus or of havingsuch an image sensing apparatus built in, such as an informationprocessing apparatus equipped with an image sensing capability (forexample a personal computer) or a portable terminal equipped with animage sensing capability (for example a mobile phone).

First Embodiment

A description is first given of the configuration of the image sensingapparatus.

FIG. 1 is a block diagram showing an example of the configuration of adigital still camera (hereinafter simply “digital camera”) according toone embodiment of the present invention.

A digital camera 100 is configured so as to sense a subject through anoptical system (an image sensing lens) 10. The optical system can beconfigured as a zoom lens, by which arrangement an optical zoom functionis provided. Furthermore, the digital camera 100 can be configured so asto have an electronic zoom function, by which an image that is sensed byan image-sensing device 14 is enlarged by image processing. It should benoted that the digital camera 100 is sometimes configured so as to haveone of either an optical zoom or an electronic zoom. Moreover, theoptical system 10 may be interchangeable, as seen for example in singlelens reflex digital cameras and the like.

The zoom function is accomplished by driving a lens used to change themagnification of the optical system 10, using a drive mechanism in theoptical system 10 itself or a drive mechanism in the body of the digitalcamera 100.

Rays of light from the subject that pass through the optical system (theimage sensing lens) 10 (that is, light rays that enter the angle of viewof the lens) pass through an opening in a shutter equipped with anaperture function to form an optical image of the subject on animage-sensing surface of the image-sensing device 14 (which may be a CCDsensor, for example, or a CMOS sensor). The image-sensing device 14converts this optical image into an analog image signal and outputs itto an A/D converter 16, which converts the analog image signal providedby the image-sensing device 14 into a digital image signal (digitaldata). The image-sensing device 14 and the A/D converter 16 arecontrolled by clock signals and control signals provide by a timinggenerator circuit 18. The timing generator circuit 18 is itselfcontrolled by a memory control circuit 22 and a system control circuit50.

An image processing circuit 20 performs certain types of imageprocessing, such as pixel interpolation and color conversion, on imagedata provided from the A/D converter 16 or image data from the memorycontroller 22. In addition, based on the sensed image data, the imageprocessing circuit 22 computes data for performing TTL (through thelens) auto focus (AF), auto exposure (AE), and pre-flash (EF), andprovides the results of those computations to the system control circuit50. The system control circuit 50 uses those computation results tocontrol an exposure controller 40 and a ranging controller (AFcontroller) 42 to achieve the auto exposure and auto focus functions.Furthermore, the image processing circuit 20 also executes a TTL autowhite balance (AWB) process based on the sensed image data.

The memory control circuit 22 controls the A/D converter 16, the timinggenerator circuit 18, the image processing circuit 20, an image displaymemory 24, a D/A converter 26, a memory 30 and a compression/expansioncircuit 32.

Image data output from the A/D converter 16 is written to the imagedisplay memory 24 or to the memory 30 either through the imageprocessing circuit 20 and the memory controller 22 or through the memorycontroller 22 alone. It should be noted that, when writing image data tothe image display memory 24, that image data is thinned as appropriatedepending on the resolution of the display of the image display unit 28.Image data for display that is written to the image display memory 24 isconverted back into an analog signal for image display by the D/Aconverter 26 and displayed by the image display unit 28. The display ofthe image display unit 28 may be a thin-film transistor (TFT) liquidcrystal display (LCD) or the like.

An electronic viewfinder (EVF) capability is achieved by continuouslydisplaying the sensed images on the image display unit 28. The displayon the image display unit 28 can be turned ON and OFF at will bycommands from the system control circuit 50. Power consumption by thedigital camera 100 can be greatly reduced by switching the display OFF.

The memory 30 is used to store sensed still images and moving images,and is provided with enough storage capacity to store a predeterminednumber of frames of still pictures or a predetermined length of time ofmoving images. As a result, large amounts of image data can be writtento the memory 30 and at high speed, even in the case of automatic imagesensing, involving multiple still images sensed continuously, and inpanorama image sensing. In addition, the memory 30 can also be used as awork area for the system controller 50.

The compression/expansion circuit 32 compresses and expands image datausing, for example, an Adaptive Discrete Cosine Transform (ADCT) or thelike. The compression/expansion circuit 32 is configured to compress orexpand the image data stored in the memory 30 and to write the processedimage data to the memory 30.

The exposure controller 40, based on information provided by the systemcontrol circuit 50, controls the shutter 12 equipped with the aperturefunction. Moreover, the exposure controller 40 may also have a flashadjustment function linked to a flash 48. The flash 48 has the abilityto project light for flash adjustment and AF auxiliary lighting.

The ranging controller 42, based on information provided by the systemcontrol circuit 50, controls the optical system 10 focusing lens. Thezoom controller 44 controls zooming of the optical system 10. A barriercontroller 46 controls the operation of a barrier 102 that protects theoptical system 10.

The system control circuit 50 may be a CPU, for example, and controlsthe overall operation of the digital camera 100 by executing a programstored in the memory. The system control circuit 50 contains an imageprocessor 50 a that executes such processes as drawing an object thatdisplays image sensing conditions and/or the operating status of theimage sensing apparatus and the like that is displayed on the imagedisplay unit 28.

The memory 52 stores constants, variables, programs and the like for theoperation of the system controller 50, as well as object data and thelike for display of an object on the image display unit 28. Typically,the object data is recorded in the memory prior to shipment of thedigital camera 100 from the factory. The object data may be rewritten bythe user using data downloaded over an electronic communications lineafter shipment of the digital camera 100 from the factory or data storedon a recording medium and provided to the user. Alternatively, it may berewritten at a service center or the like.

For example, a display unit 54, which may be an LCD or an LED, and/or,for example, a speaker sound source 55, are configured so as to outputinformation concerning the operating status of the digital camera 100 aswell as messages and the like using character, image or voicecommunications in response to the execution of a program in the systemcontrol circuit 50. The display unit 54, which may be a single unit or aplurality of units, is located where it can be seen easily from thevicinity of a control panel 70. Moreover, a portion of the display unit54 may be located within the optical viewfinder 104.

The type of information that is displayed on the display unit 54includes single shooting/continuous shooting, self-timer, compressionrate, number of recorded pixels, number of exposures recorded, number ofexposures remaining, shutter speed, aperture setting, exposurecorrection, flash, red-eye reduction, macroshooting, buzzer setting,remaining battery power for clock, remaining battery power, error,multi-digit information, insertion/removal status of recording media 200and 210, communications IF operation, date/time, external computerconnection status, focus, image sensing preparations complete,hand-shake warning, flash charged and recording medium write operation.Any or all of this information may be displayed inside the opticalviewfinder 104.

A nonvolatile memory 56 is an electrically erasable/programmable memorysuch as an EEPROM. The object data described above may be stored in thenonvolatile memory 56.

A mode dial 60, shutter switches 62 and 64, an image display ON/OFFswitch 66, a flash setting button 68 and a control panel 70 compriseoperating unit for inputting a variety of operating instructions to thesystem control circuit 50. The operating means, for example, iscomprised of combinations of single or multiple buttons, switches,dials, touch-screen panels, visual detection pointing devices, voicerecognition devices and the like.

A concrete description will now be given of the operating unit.

The mode dial 60 switches among and sets such function modes as poweroff, automatic shooting mode, shooting mode, panorama capture mode,reproduction mode, multi-screen reproduction/erasure mode, PC connectionmode, and so forth.

The first shutter switch (SW1) 62 designates is switched ON by pressinghalfway a shutter button 310, and instructs the system control circuit50 to start such operations as the AF (auto focus) process, the AE (autoexposure) process, the AWB (automatic white balance) process, the EF(pre-flash) process, and so forth.

The second shutter switch (SW2) 64 is switched ON by pressing completelythe shutter button 310, which causes the system control circuit 50 tostart a series of processes (sensing) including the following processes:After the image signal is read from the image-sensing device 14 andconverted into image data by the A/D converter 16, the image data isprocessed by the image processing circuit 20 and written to the memory30 via the memory control circuit 22. Then, the image data is read fromthe memory 30, compressed by the compression/expansion circuit 32 andthe compressed image data is then written to the recording media 200 or210.

A zoom controller 65 is used by a photographer to change the angle ofview (zoom magnification or sensing magnification), and is comprised,for example, of a slide-type control member or a lever-type controlmember and a switch or a sensor that detects operation of the controlmember.

The image display ON/OFF switch 66 controls the display ON/OFF settingof the image display unit 28. When photographing using the opticalviewfinder 104, power consumption can be reduced by turning the imagedisplay unit 28 display OFF, thereby cutting off the supply of electriccurrent.

A flash setting button 68 is used to set/change the operating mode ofthe flash. In the present embodiment, the flash setting button 68 can beset to an auto mode, a flash mode, a red-eye reduction mode and a flashmode with red-eye reduction. The auto mode is a mode that adjusts theflash automatically according to the brightness of the subject. In theflash mode, the flash is always fired. The red-eye reduction modeautomatically adjusts the flash according to the brightness of thesubject and at the same time also fires a red-eye reduction light. Inthe flash mode with red-eye reduction, both the red-eye reduction lightand the strobe are always fired.

The control panel 70 is composed of a variety of buttons andtouch-screen panels. Specifically, the control panel 70 may include amenu button, a set button, a macro button, a multi-screen reproductionpage change button, a single shooting/continuous shooting/self timerswitching button, menu navigation+(plus) button, menu navigation−(minus)button. The control panel 70 may further include a reproduction imagenavigation+(plus) button, a reproduction image−(minus) button, a picturequality selection button, an exposure correction button, a date/time setbutton, and a compression mode switch and the like.

The compression mode switch is used to set and to select the compressionrate of JPEG (Joint Photographic Expert Group) compression, recording ina RAW mode, and so forth. In the RAW mode, the analog image signal thatthe image-sensing device 14 outputs is recorded as is as digitized data(RAW data).

It should be noted that, in the present embodiment, the RAW data mayinclude not only A/D-converted data that has been photoelectricallyconverted by the image-sensing device but also data that has beenreversibly compressed after A/D conversion. Moreover, RAW data alsomeans data of a state in which the information that is output from theimage-sensing device is maintained without loss. For example, theA/D-converted analog image signal includes at least a stage in which theanalog image signal has not yet been subjected to white balanceprocessing, a stage at which the analog image signal has not yet beensubjected to color separation processing that separates the analog imagesignal into a luminance signal and a color signal, and a stage at whichthe analog image signal has not yet been subjected to colorinterpolation processing. In addition, RAW data may include not onlythat which has been digitized but also the analog image signal itself asobtained from the image-sensing device.

In the present embodiment, the JPEG compression mode includes multiplemodes, for example, a normal mode and a fine mode. The normal mode maybe selected by the digital camera 100 user if the data size of thephotographed image is of primary importance, and the fine mode may beselected by the digital camera 100 user if the picture quality of thephotographed image is of primary importance.

In the JPEG compression mode, the compression/expansion circuit 32 readsfrom the memory 30 the image data that is written to the memory 30 andrecords it on, for example, recording medium 200, after compressing theimage data at a set compression rate.

In the RAW mode, the analog image signal is read out as is line by linedepending on the layout of the pixels of a color filter of theimage-sensing device 14 image data written to the memory 30 via the A/Dconverter 16 and the memory control circuit 22 is recorded on therecording media 200 or 210.

It should be noted that the digital camera 100 of the present embodimentmay have a simultaneous multi-exposure recording mode that permitsmultiple image data recording in response to a single operationperformed by the user. For example, so-called auto bracket mode in whichmultiple image data are sensed with sensing conditions such as whitebalance and exposure have been changed in steps, is included in such amode. Recording of images generated by applying different imageprocessing to the same sensed image is also included in such mode. Suchrecording includes, for example, as described with respect to recordingin the JPEG format and in the RAW format, a plurality of image datarecordings of different data formats, image data recordings of the sameformat but different compression rates, and recordings of image datasubjected to predetermined image processing as well as image data thatis not subjected to predetermined image processing.

A power source controller 80 is comprised, for example, of a powersource detection circuit, a DC-DC converter, a switching circuit forswitching the blocks to be supplied with power. The power supplycontroller 80 also detects whether or not a power source has beeninstalled, the type of power source, and the remaining battery power,controls the DC-DC converter based on the results of those detectionoperations and on instructions from the system control circuit 50, andsupplies the necessary electrical voltage for the required time to thenecessary blocks. A power source 86 may, for example, be a primarybattery such as an alkaline battery or a lithium battery, a secondarybattery such as a NiCd battery, a NiMH battery or a Li battery, an ACadapter or the like. The body of the digital camera 100 and the powersource 86 are connected to each other by their respective connectors 82,84.

The recording media 200 and 210 are comprised of recording parts 202 and212 composed of semiconductor memories, magnetic disks or the like, aswell as interfaces 204 and 214 and connectors 206 and 216 for connectingto the digital camera 100. The recording media 200 and 210 are loadedinto the digital camera 100 through the media-side connectors 206 and216 and the digital camera 100-side connectors 92 and 96. The interfaces90 and 94 are then connected to the connectors 92, 96. Whether or notthe recording media 200 and 210 are installed is detected by a recordingmedium insertion/removal sensor 98.

It should be noted that although in the present embodiment the digitalcamera 100 has two sets of interfaces and two sets of connectors forinstalling the recording media, there may be any number of sets ofinterfaces and connectors, including one each. Moreover, the sets ofinterfaces and connectors may have different specifications.

Anything that conforms to a PCMCIA card or CF (Compact Flash (registeredtrademark)) card specification, for example, can be used as theinterfaces 90 and 94 and the connectors 92 and 96. In this case, byconnecting various communications cards, image data and managementinformation attached to the image data can be sent back and forthbetween the digital camera 100 and peripheral devices such as othercomputers, printers and the like. As the communications card there is,for example, a LAN card, a modem card, a USB card, an IEEE 1394 card, aP1284 card, a SCSI card, a PHS communications card and the like.

The optical viewfinder 104 may be a TTL viewfinder, for example, andfocuses beams of light passing through the lens 10 using prisms, mirrorsand so forth. Using the optical viewfinder 104 enables a shootingoperation without using the electronic viewfinder function provided bythe image display unit 28. Display elements that form a portion of theimage display unit 28 may be positioned within the optical viewfinder104, for example, display elements for displaying the focus, thehand-shake warning, the flash charged, the shutter speed, the aperturevalue, and the exposure correction display and the like.

A communications circuit 110 implements a variety of communicationsfunctions such as USB, IEEE 1394, P 1284, SCSI, modem, LAN, RS 232C,wireless communications and the like. A connector 112 for connecting thedigital camera 100 to other devices may be connected to thecommunications circuit 110, as may an antenna when providing wirelesscommunications.

FIG. 2 is a perspective view showing one external configuration of thedigital camera 100. It should be noted that, in FIG. 2, parts notnecessary to the description are omitted. The control panel 70 describedabove includes the buttons and switches 301-311 shown for example inFIG. 2. The user manipulates these buttons and switches 301-311 whenswitching the digital camera 100 power source ON/OFF, setting orchanging sensing conditions, checking sensing conditions, checking theoperating status of the apparatus, checking the sensed image and soforth.

A power button 311 activates and deactivates the digital camera 100, orswitches the digital camera 100 main power source ON/OFF. A menu button302 displays a setting menu for the sensing conditions, operating modeand the like of the digital camera 100, and displays a menu fordisplaying the operating status of the digital camera 100. The menus mayhave a layered structure, for example, with each layer containingselectable items and/or items whose values can be changed.

An erase button 301, when pressed, erases images displayed in an imagelist mode or on a sensed image confirmation screen. In the presentembodiment, the sensed image confirmation screen displays the sensedimage on the image display unit 28 immediately after sensing to permitthe user to check the results (a so-called quick preview screen). In thepresent embodiment, the sensed image confirmation screen is displayedwhen the user presses the shutter button 310 to instruct sense and thenholds the shutter button 310 down after sensing, for as long as theshutter button 310 is pressed and held down.

A setting button 303, when pressed, sets or selects a mode or an item.When the setting button 303 is pressed, the system control circuit 50sets the mode or the item selected at the time. The image display ON/OFFswitch 66 is used to select display/non-display of sensing informationfor a sensed image and to switch between having the image display unit28 function as an electronic viewfinder or not.

A left button 305, a right button 306, an up button 307 and a downbutton 308 may be used for such purposes as changing a choice (forexample, an item or an image) selected from among a plurality of choicesby a cursor or by highlighting or the like, changing the positions ofindices that define selected menu choices, and increasing or decreasingnumerical values (for example, numerical values indicating correctionvalues, date/time, or the like).

It should be noted that, in the present embodiment, a color conversioneffect that changes the color of a particular area contained within asensed image is used as an example of an image process (that is, aneffect) that can be applied to a sensed image. Designation of the targetcolor to be changed (the source color) and the color that is to result(the destination color) is carried out using an image displayed in theelectronic viewfinder.

Specifically, in a color conversion effect setting screen (a colordesignation screen), pressing the left button 305 sets an average colorof an image contained in a small area in the center of the electronicviewfinder as the source color and pressing the right button 306similarly sets the destination color. Of course, any other method may beused to designate the source color and the destination color, such as,for example, using dedicated buttons without enlisting directionalbuttons, designating the source color and the destination color oneafter the other using the same button, and so forth.

When pressed halfway, for example, the shutter button 310 instructs thesystem control circuit 50 to start the AF (auto focus) process, the AE(auto focus) process, the AWB (auto white balance) process, the EF(pre-flash) process and the like. When pressed completely, the systemcontrol circuit 50 is instructed to sense an image.

The zoom controller 65, as described above, is operated by aphotographer to change the angle of view (zoom magnification or sensingmagnification).

A record/reproduce switching switch 312 is used to switch the recordingmode to the reproduction mode as well as to switch the reproduction modeto the recording mode. It should be noted that, in place of theforegoing control system, a dial switch or some other control system maybe used instead.

A description will now be given of the operation of the image sensingapparatus of the present invention.

FIG. 3 is a flow chart illustrating a sensing operation of the digitalcamera 100 according to the present embodiment in case that an effect isto be applied to a sensed image. In the present embodiment, as an imageprocess that can be applied during image sensing operation (a sensingeffect), a description is given of a case in which a color conversionprocess that converts a target color contained in a sensed image into adestination color is selected. Moreover, execution of a shooting modeapplying a color conversion process is set by the user in advancethrough the setting menu.

First, in step S101, the memory control circuit 22 causes the imagedisplay unit 28 to function as an electronic viewfinder by continuouslywriting image data output by the A/D converter 16 to the image displaymemory 24. Moreover, the image processor 50 a inside the system controlcircuit 50 writes graphical user interface (GUI) data for enabling theuser to designate the source color and the destination color that arenecessary to the color conversion process to the image display memory24. These processes allow the color designation screen to be displayedon the image display unit 28. Thus, the color designation screenconsists of an ordinary electronic viewfinder to which a special GUI isadded. It should be noted that, in the present embodiment, the imagesthat are displayed in real time so as to achieve the electronicviewfinder are also called through images.

FIG. 4 is a diagram showing an example of the color designation screen.A color designation screen 800 is broadly divided into an electronicviewfinder (EVF) area (hereinafter called simply “EVF”) 801 and a GUIarea 802. A fixed color designation frame 803 is displayed in the centerof the electronic viewfinder area 801. A change source color displayframe 804 and a change destination color display frame 805,respectively, are provided in the GUI area 802. Moreover, indicators 806and 807 are provided so that the user can easily grasp visually that thechange source color (Src) designation function is allocated to the leftbutton and the change destination color (Dst) designation function isallocated to the right button, respectively.

Here, the user is attempting to change the colors of a potted plantcurrently displayed in the EVF 801 using the color-conversion process.In this case, in order to designate the change source color, the userfirst either approaches the potted plant or uses the optical zoomfunction to place the petals of the plant substantially in the center ofthe color designation frame 803 and then presses the left button 305 asthe source color designation button.

The system control circuit 50 detects when the source color designationbutton in the color designation screen is pressed (step S103), reads theimage data corresponding to the coordinates within the color designationframe 803 image display memory 24, and obtains the source color.Although there is no particular limitation on the method used to obtainthe source color from the image data within the color designation frame803, it is possible to use a color that corresponds to the average valueof the image data within the color designation frame 803. The systemcontrol circuit 50 sets the source color thus obtained by saving it tothe memory 30 and also displays it in the source color display frame 804of the GUI area 802 through the image processor 50 a(step S105). Thiscondition is shown in FIG. 5.

In step S109, the system control circuit 50 checks if the destinationcolor has been set. Here, the destination color has not yet been set,and thus processing returns to step S103. At this time, on the throughimage, the area of the color that corresponds to the source color may beshaded with diagonal stripes or overlaid by a grid for easyidentification. In the present embodiment, as with the designation ofthe source color, the designation of the destination color is carriedout using the through image.

FIG. 6 shows a state in which the user is attempting to designate thedestination color from a background area on the color designationscreen. In step S111 the system control circuit 50 detects when the userpresses the right button 806, which is the destination color designationbutton, in such a state.

Then, the system control circuit 50 obtains the destination color usingthe same method as that used to obtain the source color and sets thedestination color by storing it in the memory 30. At the same time, thesystem control circuit 50 displays the destination color in thedestination color display frame 805 of the GUI area 802 through theimage processor 50 a (step S113).

Next, in step S117 the system control circuit 50 performs a previewprocess.

If the shutter button is pressed during preview (step S121) the imageprocessing circuit 20 performs a shooting process to which the colorconversion process is applied (step S123), and records the sensed imageon the recording media 200 or 210.

By contrast, if the source color designation button or the destinationcolor designation button is pressed once again during preview, thesource color or the destination color is reset based on the image datainside the color designation frame of the through image at that time.

Next, a detailed description is given of the preview process in stepS117, using the flow chart shown in FIG. 7.

First, in a step S1701, the system control circuit 50 instructs theimage processing circuit 20 to apply a color conversion process to thethrough image. Then, in step S1703, the system control circuit 50performs an electronic viewfinder display process. Implementation of thecolor conversion process is set in step S1701, and therefore the imageprocessing circuit 20 performs a color conversion process on the imagedata output by the A/D converter 16 using the source color and thedestination color stored in the memory 30 and supplies the colorconversion processed image data to the memory controller 22. The memorycontroller 22 then writes the color conversion processed image data tothe image display memory 24. Thus the color designation screen changesfrom the state of the through image before color conversion processingshown in FIG. 6 to the state shown in FIG. 8, in which the color of theflower petals has been changed.

It should be noted that although it is possible to implement this typeof color conversion process using well known image processingtechnology, it is preferable to use a color conversion process that usesa three-dimensional look-up table (LUT) like that disclosed in JapaneseLaid-Open Patent Publication No. 2004-80100, for example, because doingso maintains tonality.

Next, the system control circuit 50 checks if the shutter button hasbeen pressed or not (step S1705). If the shutter button has beenpressed, then processing proceeds to step S121 (FIG. 3). If the shutterbutton has not been pressed, then in step S1707 the passage of apredetermined period of time (for example, 1 second) is awaited. Whenthe predetermined period of time elapses, in step S1709 the systemcontrol circuit 50 inverts the setting of image processing circuit 20for the application of the color conversion process to the throughimage. In other words, at this point the image processing circuit 20 isset both to not apply the color conversion process, as well as to applythe color conversion process when the predetermined period of time nextelapses.

Thus, as described above, a state of flashing display of the throughimage prior to color conversion and the preview image after conversionis achieved. The order of flashing is set so that the through imageprior to color conversion is displayed first. This arrangement enablesthe user to comprehend which color of the image has actually beenchanged by comparing it to the original image.

In addition, in such a state, if the user is satisfied with thepost-color conversion preview image, the user can carry out imagesensing as is by pressing the shutter button (step S121→step S123). Ifthe user is not satisfied with the post-color conversion preview image,the user can redo the color designation (step S121→step S103). It shouldbe noted that the process of redoing the color designation by pressingthe color designation buttons once again changes the order of flashingso that the preview image is displayed first and then the through imagewith no color converse is displayed, thereby enabling the colorconversion to be redone upon checking which area of color is currentlychanging.

Thus, as described above, by instructing the application of the colorconversion process to the through image to be executed and thencancelled one after the other with each passage of a predeterminedperiod of time, a non-color-converted image (FIG. 6) and acolor-converted image (FIG. 8) are displayed one after the other on theimage display unit 28. It should be noted that a variety of methods maybe used to enable the user to distinguish clearly between thecolor-converted image and the non-color-converted image, such as usingfade-out and fade-in, not displaying an image for a brief period of timewhen switching between the two types of images, displaying flashingimages, and so forth.

In addition, in order to indicate which flashing image is the originalimage and which is the processed image, an identifying display may beadded to at least one of the original image and the processed image. Forexample, the letter “S”, for “Source”, may be superimposed on theoriginal image, in order to identify it as such.

The preview process is carried out using an electronic viewfinderdisplay process, and thus the display of the EVF 801 of the colordesignation screen 800 tracks user operation even when the user performsa zooming operation or moves the camera. Therefore, in the state shownin FIG. 8, when zooming to the wide-angle side or when the user movesthe camera away from the subject, the state shown in FIG. 4 and thestate show in FIG. 9 are displayed one after the other.

Thus, by providing a preview display of the effect that is set whenperforming image sensing while applying an image processing effect tothe sensed image, the present embodiment enables the user to confirmwhat kind of effect the set effect has on the sensed image before actualimage sensing is performed. Moreover, the preview display is performedwith the electronic viewfinder display image, enabling the user toconfirm the effect of image processing in real time prior to imagesensing.

In addition, because the preview display displays the original image onwhich image processing has not been performed and the image to whichimage processing has been applied in such a way that the two images canbe compared, the present embodiment enables the user to grasp the effectof image processing more accurately and more easily.

Second Embodiment

In the first embodiment of the present invention, the preview display isaccomplished by ordering of the execution of the application of imageprocessing and ordering the cancellation of that execution one after theother so as to switch automatically between an original image that hasnot been processed and a processed image that has been processed witheach passage of a predetermined period of time. By contrast, a secondembodiment of the present invention switches the display by ordering theexecution of the application of image processing and the cancellation ofthat execution one after the other in response to a user instruction. Itshould be noted that, other than the preview process, the operation andthe structure of a digital camera 100 according to this secondembodiment are identical to those of the first embodiment, and thereforea description is given only of the preview process in the secondembodiment.

FIG. 10 is a flow chart illustrating the preview process in the digitalcamera 100 that is the image sensing apparatus according to the secondembodiment of the present invention. In FIG. 10, processes that are thesame as the preview process in the first embodiment are given the samereference numerals.

The processes of steps S1701 and S1703 are the same as the firstembodiment. In step S1705, the system control circuit 50 checks whetheror not the shutter button has been pressed, and if so processingproceeds to step S1711. In step S1711, the system control circuit 50confirms whether or not a display switching button (for example, thedisplay ON/OFF button 66) has been pressed. Processing proceeds to stepS121 (in FIG. 3) if a button other than the display switching button hasbeen pressed.

If the display switching button has been pressed, then in step S1709 thesystem control circuit 50 inverts the setting of the image processingcircuit 20 for the application of the color conversion process to thethrough image. In other words, at this point the image processingcircuit 20 is set both to not apply the color conversion process, aswell as to apply the color conversion process when the predeterminedperiod of time next elapses.

Thus, as described above, in the present embodiment, by ordering theexecution of application of the color conversion process to the throughimage and then the cancellation of the execution of the application ofthe color conversion process to the through image each time the displayswitching button is pressed, the non-color-converted image and thecolor-converted image are displayed one after the other on the imagedisplay unit 28.

In the second embodiment, as with the first embodiment, an identifyingdisplay may be added to at least one of the original image and the imageafter image processing is applied.

Thus, as described above, by providing a preview display of the effectthat is set when performing image sensing while applying an imageprocessing effect to the sensed image, the present embodiment enablesthe user to confirm what kind of effect the set effect has on the sensedimage.

In addition, because the preview display switches the display of theoriginal image on which image processing has not been performed and theimage to which image processing has been applied by user instruction,the present embodiment enables the user to grasp the effect of imageprocessing more accurately and more easily. Moreover, unlike the firstembodiment, the second embodiment enables the user to compare theoriginal image and the processed image at a time desired by the user.

Other Embodiments

Although in the foregoing embodiments a color conversion process thatconverts one particular color into a different color is described as anexample of the type of image processing that can be designated duringimage sensing, other types of image processing are also possible.Moreover, it is not necessary that the conversion of color involve asingle pair of colors, and thus multiple sets of source colors anddestination colors may be designated.

In addition, the units that comprise the image sensing apparatus in theforegoing embodiments of the present invention as well as the steps thatimplement the image sensing method, may be implemented as software bycausing a computer (or CPU) contained in the image sensing apparatus toexecute a control program. Such a control program, as well as acomputer-readable storage medium storing the control program, are alsoincluded in the present invention.

In addition, in the foregoing embodiments a structure is described thatswitches between through images to which image processing is applied andthrough images to which image processing is not applied by switching theapplication of image processing by the image processing circuit 20 onand off. However, the present invention is not limited to such anarrangement, and any method may be used provided that it enables thedisplay to switch between a processed image and an unprocessed originalimage.

For example, the image processing circuit 20 itself constantly executesan operation that applies image processing. If the memory controller 22is then configured so as to select an image to be written to the imagedisplay memory 24 from a through image that does not pass through theimage processing circuit 20 and an image to which image processing hasbeen applied by the image processing circuit 20, then the same resultcan be achieved.

In such a case, in step S1701 shown in FIG. 7 and FIG. 10, the imagethat is to be written to the image display memory 24 is set to the imagethat the image processing circuit 20 outputs. Then, in step S1709, theimage to be written to the image display memory 24 is set to athrough-image that does not pass through the image processing circuit20.

As many apparently widely different embodiments of the present inventioncan be made without departing from the spirit and scope thereof, it isto be understood that the invention is not limited to the specificembodiments thereof except as defined in the appended claims.

This application claims the benefit of Japanese Patent Application No.2005-018523 filed on Jan. 26, 2005, which is hereby incorporated byreference herein its entirety.

What is claimed is:
 1. An image pickup apparatus comprising: an imagesensing unit that senses rays of light from a subject that pass throughan optical system and outputs an image of the subject; a display unitthat continuously displays images obtained in real time from the imagesensing unit, as an electronic view finder; an image processing unitthat applies color conversion processing, by using a look-up table, tothe image, in which color conversion processing a color of the image isconverted into another color, wherein at least one of the followingconditions is satisfied: (a) the color to be converted is determinedbased on a partial area of an image obtained from the image sensingunit, the partial area being designated by a user operation, and (b) theanother color is determined based on a partial area of an image obtainedfrom the image sensing unit, the partial area being designated by a useroperation; and a control unit that controls the display unit toalternately display a processed image to which the color conversionprocessing is applied by the image processing unit, and a non-processedimage to which the color conversion processing is not applied by theimage processing unit, so as to enable the user to compare the processedimage with the non-processed image.
 2. The image pickup apparatusaccording to claim 1, wherein the control unit controls the display unitto alternately display the processed image and the non-processed imagewith each passage of a predetermined period of time.
 3. The image pickupapparatus according to claim 1, wherein the control unit controls thedisplay unit to alternately display the processed image and thenon-processed image in response to a user operation.
 4. The image pickupapparatus according to claim 1, wherein the value for the colorconversion processing is determined using the partial area of the imagethat is obtained from the image sensing unit and resized as appropriatedepending on the resolution of the display of the display unit.
 5. Theimage pickup apparatus according to claim 1, wherein the control unitcontrols the image processing unit to apply the color conversionprocessing to an image and record the image in a memory in response to auser operation, and wherein the control unit controls the imageprocessing unit to reset the value for the color conversion processingset by the user operation in response to another user operation.
 6. Theimage processing apparatus according to claim 1, wherein in the colorconversion processing, a color in an area of the image, the areaincluding an area other than the partial area of an image obtained fromthe image sensing unit, is converted.
 7. An image processing apparatuscomprising: a display unit that continuously displays images obtained inreal time from an image generating unit, as an electronic viewfinder; animage processing unit that applies an image processing operation to animage obtained in real time from the image generating unit on the basisof a value determined based on a partial area of an image obtained fromthe image generating unit, the partial area being designated by a useroperation; and a control unit that controls the display unit to display,as the images obtained in real time, (a) a processed image to which theimage processing operation is applied by the image processing unit, and(b) a non-processed image to which the image processing operation is notapplied by the image processing unit such that the processed image andthe non-processed image are periodically and automatically switched soas to enable the user to compare the processed image with thenon-processed image.
 8. The image processing apparatus according toclaim 7, wherein the image processing operation is a color conversionprocessing operation in which a color of the image obtained by the imagegenerating unit is converted into another color on the basis of thevalue.
 9. The image processing apparatus according to claim 7, whereinthe control unit controls the display unit to alternately display theprocessed image and the non-processed image with each passage of apredetermined period of time.
 10. The image processing apparatusaccording to claim 7, wherein the control unit controls the display unitto alternately display the processed image and the non-processed imagein response to a user operation.
 11. The image processing apparatusaccording to claim 7, wherein the value for the image processingoperation is determined using the partial area of the image that isobtained from the image generating unit and resized as appropriatedepending on the resolution of the display of the display unit.
 12. Theimage processing apparatus according to claim 11, wherein the imageprocessing operation comprises a color conversion processing operationin which a color of the image obtained by the image generating unit isconverted into another color on the basis of the value.
 13. The imageprocessing apparatus according to claim 7, wherein the control unitcontrols the image processing unit to apply the image processingoperation to an image and record the image in a memory in response to auser operation, and the control unit controls the image processing unitto reset the value for the image processing operation in response toanother user operation.
 14. The image processing apparatus according toclaim 13, wherein the image processing operation comprises a colorconversion processing operation in which a color of the image obtainedby the image generating unit is converted into another color on thebasis of the value.
 15. An image processing method comprising: an imagesensing step of sensing, by an image sensing unit, rays of light from asubject that pass through an optical system and outputting an image ofthe subject; a display step of continuously displaying the imagesobtained in real time from the image sensing unit on a display unit asan electronic view finder; an image processing step of applying colorconversion processing, by using a look-up table, to the image, in whichcolor conversion processing a color of the image is converted intoanother color, wherein at least one of the following conditions issatisfied: (a) the color to be converted is determined based on apartial area of an image obtained from the image sensing unit, thepartial area being designated by a user operation, and (b) the anothercolor is determined based on a partial area of an image obtained fromthe image sensing unit, the partial area being designated by a useroperation; and a control step of controlling the display step toalternately display a processed image to which the color conversionprocessing is applied in the image processing step, and a non-processedimage to which the color conversion processing is not applied in theimage processing step, so as to enable the user to compare the processedimage with the non-processed image.
 16. The image processing methodaccording to claim 15, wherein in the color conversion processing, acolor in an area of the image, the area including an area other than thepartial area of an image obtained from the image sensing unit, isconverted.
 17. An image processing method comprising: a display step ofcontinuously displaying images obtained in real time from an imagegenerating unit on a display unit as an electronic viewfinder; an imageprocessing step of applying an image processing operation to an imageobtained in real time from the image generating unit on the basis of avalue determined based on a partial area of an image obtained from theimage generating unit, the partial area being designated by a useroperation; and a control step of controlling the display step todisplay, as the images obtained in real time, (a) a processed image towhich the image processing operation is applied by the image processingstep, and (b) a non-processed image to which the image processingoperation is not applied in the image processing step such that theprocessed image and the non-processed image are periodically andautomatically switched so as to enable the user to compare the processedimage with the non-processed image.